The approaches by which the machine learning and clinical research communities utilize real world data (RWD), including data captured in the electronic health record (EHR), vary dramatically. While clinical researchers cautiously use RWD for clinical investigations, ML for healthcare teams consume public datasets with minimal scrutiny to develop new algorithms. This study bridges this gap by developing and validating ML-DQA, a data quality assurance framework grounded in RWD best practices. The ML-DQA framework is applied to five ML projects across two geographies, different medical conditions, and different cohorts. A total of 2,999 quality checks and 24 quality reports were generated on RWD gathered on 247,536 patients across the five projects. Five generalizable practices emerge: all projects used a similar method to group redundant data element representations; all projects used automated utilities to build diagnosis and medication data elements; all projects used a common library of rules-based transformations; all projects used a unified approach to assign data quality checks to data elements; and all projects used a similar approach to clinical adjudication. An average of 5.8 individuals, including clinicians, data scientists, and trainees, were involved in implementing ML-DQA for each project and an average of 23.4 data elements per project were either transformed or removed in response to ML-DQA. This study demonstrates the importance role of ML-DQA in healthcare projects and provides teams a framework to conduct these essential activities.

It's 10 a.m. on a Monday, and Aman, one of the developers of a new artificial intelligence tool, is excited about the technology launching that day. Leaders of Duke University Hospital's intensive care unit had asked Aman and his colleagues to develop an AI tool to help prevent overcrowding in their unit. Research had shown that patients coming to the hospital with a particular type of heart attack did not require hospitalization in the ICU, and its leaders hoped that an AI tool would help emergency room clinicians identify these patients and refer them to noncritical care. This would both improve quality of care for patients and reduce unnecessary costs. Aman and his team of cardiologists, data scientists, computer scientists, and project managers had developed an AI tool that made it easy for clinicians to identify these patients.

Artificial intelligence tools are complicated computer programs that suck in vast amounts of data, search for patterns or trajectories, and make a prediction or recommendation to help guide a decision. Patients don't need to understand these algorithms at a data-scientist level, but it's still useful for people to have a general idea of how AI-based healthcare tools work, says Suresh Balu, program director at the Duke Institute for Health Innovation. Some patients can get a little jumpy when they hear algorithms are being used in their care, says Mark Sendak, a data scientist at the Duke Institute for Health Innovation. We picked an algorithm that flags patients in the early stages of sepsis -- a life-threatening complication from an infection that results in widespread inflammation through the body. The algorithm we're looking at underpins a program called Sepsis Watch, which Sendak and Balu helped develop at Duke University.

Artificial intelligence algorithms are everywhere in healthcare. They sort through patients' data to predict who will develop medical conditions like heart disease or diabetes, they help doctors figure out which people in an emergency room are the sickest, and they screen medical images to find evidence of diseases. But even as AI algorithms become more important to medicine, they're often invisible to people receiving care. Artificial intelligence tools are complicated computer programs that suck in vast amounts of data, search for patterns or trajectories, and make a prediction or recommendation to help guide a decision. Sometimes, the way algorithms process all of the information they're taking in is a black box -- inscrutable even to the people who designed the program.

In hospitals, doctors and nurses keep vigilant watch over patients' vital signs and blood tests to catch the first symptoms of sepsis. In this life-threatening condition, the body responds to an infection with widespread inflammation that can lead to organ failure. Cases can progress rapidly to severe sepsis and then to septic shock, which has a mortality rate of almost 50 percent in the United States. But even the most vigilant humans get tired, make mistakes, and miss subtle patterns. That's why several hospitals are experimenting with artificially intelligent sepsis detectors.